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1 . A method for clustering data comprising the steps of 

a. constructing a graph of a database in which each node of the graph 
represents a component part of the database, and every two nodes represent 
neighboring component parts associated by an arc representing a coupling value, 

b. selecting chosen component parts as blocks with unselected neighboring 
component parts coupled with a selected block according to coupling values, 

c. coarsening the graph by performing iterated weighted "aggregation wherein at 
each iteration of the coarsening blocks are selected and coupling values updated 
between unselected blocks to form larger blocks to obtain hierarchical decomposition 
of the database and to form a pyramid structure, 

d. adjusting the coupling between blocks. 

e. detecting saliency of segments in the pyramidal structure, 

f. determining which component parts belong to a segment by computing 
recursively a degree of attachment of every component part to each of the blocks in 
the pyramid, and 

g. scanning the pyramid from coarse to fine starting at the level a segment is 
detected and applying relaxation sweeps to sharpen the boundaries of a segment. 

2. A method for processing an image comprising the steps of 

a. constructing a graph of an image in which each node of the graph represents 
a pixel of the image, and every two nodes represent neighboring pixels associated 
by an arc representing a coupling value, 

b. selecting chosen pixels as blocks with unselected neighboring pixels coupled 
with a selected block according to coupling values, 

c. coarsening the graph by performing iterated weighted aggregation wherein at 
each iteration of the coarsening blocks are selected and coupling values updated 
between unselected blocks to form larger blocks 

to obtain hierarchical decomposition of the database and to form a pyramid 
structure,. 

d. adjusting the coupling between blocks. 

e. detecting saliency of segments in the pyramidal structure, 

f. determining which pixels belong to a segment by computing recursively a 
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degree of attachment of every pixel to each of the blocks in the pyramid, and 
g- scanning the pyramid from coarse to fine starting at the level a segment is 
detected and applying relaxation sweeps to sharpen the boundaries of a segment. 

3. A method for processing an image comprising the steps of 

a. constructing a graph of an image in which each node of the graph represents 
a pixel of the image, every edge connects a pair of neighboring pixels and a weight 
is associated with each edge reflecting contrast in the corresponding location in the 
image, 

b. selecting some pixels as blocks and associating unselected neighboring 
pixels with a selected block to form aggregates, 

c. detecting segments by a recursive coarsening using weighted aggregation 
which induces a pyramid structure over the image, the segments detected appearing 
as an aggregate at some level in the pyramid, 

d. said recursive coarsening comprising iterated weighted aggregation wherein 
at each iteration of the coarsening blocks are selected and weights are updated 
between unselected blocks to form larger blocks to obtain hierarchical decomposition 
of the image into aggregates, 

e. determining salient segments from among the segments detected in the 
pyramidal structure, and 

f. sharpening the segments to determine its boundaries more accurately. 

4. The method according to claim 3 including the further step of determining which 
pixels belong to a segment by computing recursively a degree of attachment of every 
pixel to each of the blocks in the pyramid. 

5. The method according to claim 3 including the further step of scanning the 
pyramid from coarse to fine starting at the level a segment is detected and applying 
relaxation sweeps to sharpen the boundaries of a segment. 

6. The method according to claim 3 including the further step of computing at least 
one property of the aggregate it represents during recursive coarsening for every 
block in a new level of the pyramid. 
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7. The method according to claim 6 including the further step of updating weights 
to account for properties computed during recursive coarsening for every edge in the 
graph. 

8. The method according to claim 3 including the further step of updating weights 
during cursive coarsening to increase weights between neighboring aggregates 
exhibiting sharp sections that connect by a smooth curve. 

9. The method according to claim 3 including the further step of applying a top- 
down sharpening during the recursive coarsening at any given level by first going 
down a preselected number of levels to check the boundaries of detected segments, 
updating weights, and rebuilding the pyramid to the originating level to sharpen 
distinction between aggregates before building the pyramid to the next upper level. 

10. The method according to claim 3 including the further step of going back down a 
preselected number of levels to check sub-aggregates regarding boundaries, update 
and rebuild the pyramid before proceeding to the next upper level as part of each 
iteration of weight aggregation. 

1 1 . The method according to claim 3 including the further step of detecting sharp 
transitions in pixels in the image. 

12. The method according to claim 3 including the further step of establishing a 
threshold to determine edge pixels in the image. 

13. The method according to claim 12 including the further step of applying edge 
tracing by best fitting line segments of aggregates to determined edge pixels. 

14. The method according to claim 13 including the further step of producing a 
polygonal approximation of an aggregate's boundary. 

15. The method according to claim 3 including the further step of comparing the 
properties of neighboring aggregates. 

16. The method according to claim 3 including the further step of modifying weights to 
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control intensity contrast between aggregates during recursive coarsening. 



17. The method according to claim 3 including the further step of determining 
variance of an aggregate relative to a neighboring aggregate. 

18. The method according to claim 3 including the further step of determining 
multiscale variance of an aggregate to detect its texture. 

19. The method according to claim 3 including the further step of determining 
average variance at finer scales to determine a relationship between aggregates. 

20. Apparatus for clustering data comprising a computer processor programmed 

a. for constructing a graph of a database in which each node of the graph 
represents a component part of the database, and every two nodes represent 
neighboring component parts associated by an arc representing a coupling value, 

b. to select chosen component parts as blocks with unselected neighboring 
component parts coupled with a selected block according to coupling values, 

c. to coarsen the graph by performing iterated weighted aggregation wherein at 
each iteration of the coarsening blocks are selected and coupling values updated 
between unselected blocks to form larger blocks to obtain hierarchical decomposition 
of the database and to form a pyramid structure, 

d. to adjust the coupling between blocks. 

e. to detect saliency of segments in the pyramidal structure, 

f. to determine which component parts belong to a segment by computing 
recursively a degree of attachment of every component part to each of the blocks in 
the pyramid, and 

g. for scanning the pyramid from coarse to fine starting at the level a segment is 
detected and applying relaxation sweeps to sharpen the boundaries of a segment. 

21 . Apparatus for processing an image comprising a computer processor programmed 

a. for constructing a graph of an image in which each node of the graph 
represents a pixel of the image, and every two nodes represent neighboring pixels 
associated by an arc representing a coupling value, 

b. for selecting chosen pixels as blocks with unselected neighboring pixels 
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coupled with a selected block according to coupling values, 

c. for coarsening the graph by performing iterated weighted aggregation wherein 
at each iteration of the coarsening blocks are selected and coupling values updated 
between unselected blocks to form larger blocks to obtain hierarchical decomposition 
of the database and to form a pyramid structure, 

d. for adjusting the coupling between blocks. 

e. for detecting saliency of segments in the pyramidal structure, 

f. for determining which pixels belong to a segment by computing 
recursively a degree of attachment of every pixel to each of the blocks in 
the pyramid, and 

g. for scanning the pyramid from coarse to fine starting at the level a segment is 
detected and applying relaxation sweeps to sharpen the boundaries of a segment. 

22. Apparatus for processing an image comprising a computer processor 
programmed 

a. for constructing a graph of an image in which each node of the graph 
represents a pixel of the image, every edge connects a pair of neighboring pixels 
and a weight is associated with each edge reflecting contrast in the corresponding 
location in the image, 

b. for selecting some pixels as blocks and associating unselected neighboring 
pixels with a selected block to form aggregates, 

c. for detecting segments by a recursive coarsening using weighted aggregation 
which induces a pyramid structure over the image, the segments detected appearing 
as an aggregate at some level in the pyramid, 

d. said recursive coarsening comprising iterated weighted aggregation wherein 
at each iteration of the coarsening blocks are selected and weights are updated 
between unselected blocks to form larger blocks to obtain hierarchical decomposition 
of the image into aggregates, 

e. for determining salient segments from among the segments detected in the 
pyramidal structure, and 

f. for sharpening the segments to determine its boundaries more accurately, 

23. Apparatus according to claim 22 wherein the computer processor is further 
programmed for determining which pixels belong to a segment by computing 
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recursively a degree of attachment of every pixel to each of the blocks in the 
pyramid. 



24. Apparatus according to claim 22 wherein the computer processor is further 
programmed for scanning the pyramid from coarse to fine starting at the level a 
segment is detected and applying relaxation sweeps to sharpen the boundaries of a 
segment. 

25. Apparatus according to claim 22 wherein the computer processor is further 
programmed for computing at least one property of the aggregate it represents 
during recursive coarsening for every block in a new level of the pyramid. 

26. Apparatus according to claim 25 wherein the computer processor is further 
programmed for updating weights to account for properties computed during 
recursive coarsening for every edge in the graph. 

27. Apparatus according to claim 22 wherein the computer processor is further 
programmed for updating weights during recursive coarsening to increase weights 
between neighboring aggregates exhibiting sharp sections that connect by a smooth 
curve. 

28. Apparatus according to claim 22 wherein the computer processor is further 
programmed for applying a top-down sharpening during the recursive coarsening at 
any given level by first going down a preselected number of levels to check the 
boundaries of detected segments, updating weights, and rebuilding the pyramid to 
the originating level to sharpen distinction between aggregates before building the 
pyramid to the next upper level. 

29. Apparatus according to claim 22 wherein the computer processor is further 
programmed for going back down a preselected number of levels to check sub- 
aggregates regarding boundaries, update and rebuild the pyramid before proceeding 
to the next upper level as part of each iteration of weight aggregation. 

30. Apparatus according to claim 22 wherein the computer processor is further 
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programmed for detecting sharp transitions in pixels in the image. 



31 . Apparatus according to claim 22 wherein the computer processor is further 
programmed for establishing a threshold to determine edge pixels in the image. 

32. Apparatus according to claim 31 wherein the computer processor is further 
programmed for applying edge tracing by best fitting line segments of aggregates to 
determined edge pixels. 

33. Apparatus according to claim 32 wherein the computer processor is further 
programmed for producing a polygonal approximation of an aggregate's boundary. 

34. Apparatus according to claim 22 wherein the computer processor is further 
programmed for comparing the properties of neighboring aggregates. 

35. Apparatus according to claim 22 wherein the computer processor is further 
programmed for modifying weights to control intensity contrast between aggregates 
during recursive coarsening. 

36. Apparatus according to claim 22 wherein the computer processor is further 
programmed for determining variance of an aggregate relative to a neighboring 
aggregate. 

37. Apparatus according to claim 22 wherein the computer processor is further 
programmed for determining multiscale variance of an aggregate to detect its texture. 

38. Apparatus according to claim 22 wherein the computer processor is further 
programmed for determining average variance at finer scales to determine a 
relationship between aggregates. 

39. A method for finding correspondence between portions of two images 
comprising the steps of 

a) subjecting the two images to segmentation by weighted aggregation, 

b) constructing directed acylic graphs from the output of the segmentation by 
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weighted aggregation to obtain hierarchical graphs of aggregates, and 

c) applying a maximally weighted subgraph isomorphism to the hierarchical 
graphs of aggregates to find matches between them. Two algorithms are 
described, One seeks a one-to-one matching between regions. The other 
computes a soft matching that is an aggregate may have more than one 
corresponding aggregate. 

d) Recovering epipolar lines and camera motion using such correspondences. 

40. Apparatus for finding correspondence between portions of two images 
comprising: 

means for subjecting the two images to segmentation by weighted aggregation to 

obtain full multiscale pyramidal representations of the images, 

means for constructing directed acylic graphs from the full multiscale pyramidal 

representations of the images to obtain hierarchical graphs of aggregates, 

means for applying a maximally weighted subgraph isomorphism to the hierarchical 

graphs of aggregates to find matches between them using an algorithm that matches 

between regions, and 

means for recovering epipolar lines and camera motion using such 
correspondences. 

41 . Apparatus according to claim 40, wherein the algorithm computes a soft 
matching such that an aggregate may have more than one corresponding aggregate. 

42. Apparatus according to claim 40, wherein the algorithm computes a one-to-one 
matching between regions 

43. A method for finding correspondence between portions of two images comprising 
the steps of: 

e) subjecting the two images to segmentation by weighted aggregation 
employing a series of coarsening in successively coarser levels, 

f) modifying the weights in each successive level to incorporate coarser 
measures of difference between neighboring aggregates based on a 
measure of difference between their average intensities and by a measure 
reflecting their motion profiles, and 
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g) recovering at the highest level a representation of the correspondence 
between the portion of the two images. 



44. A method according to claim 43 wherein the weight is determined according to 

where I t and I } denote the intensities of the two neighboring pixels, and fi is 
a positive constant, and a measure wf* reflecting the difference in the motion 
profiles associated to the two pixels. 

45. Apparatus for finding correspondence between portions of two images 
comprising: 

a. means for subjecting the two images to segmentation by weighted 
aggregation employing a series of coarsening in successively coarser levels, 

b. means for modifying the weights in each successive level to incorporate 
coarser measures of difference between neighboring aggregates based on a 
measure of difference between their average intensities and by a measure reflecting 
their motion profiles, and 

c. means for recovering at the highest level a representation of the 
correspondence between the portion of the two images. 

46. Apparatus according to claim 45 further including: 

means for determining the weight according to 

where I i and Ij denote the intensities of the two neighboring pixels, and is a 
positive constant, and a measure wff reflecting the difference in the motion profiles 
associated to the two pixels. 

47. A method for finding correspondence between portions of two images 
comprising the steps of: 

a) select two images km and Jm z . 

b) prepare for each pixel in inn a motion profile, 

c) assign a weight to each pair of neighboring pixels according to a 
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normalized correlation between their motion profiles, 

d) perform a coarsening iteration by 

e) clustering and re-estimation wherein clustering is achieved by 
selecting a set of seeds such that the remaining elements are strongly 
connected to this set, and defining the strength of association of a fine 
element / to a coarse seed k, and wherein re-estimation is achieved for each 
seed by determining the motion profile of the seed by multiplying the motion 
profiles of its children, determining whether the seed is peaked or bar-peaked, 
accumulating adaptively, moments (orders one to four) originated by peaked 
seeds, and accumulating separately, moments (orders one and two) by bar- 
peaked seeds, and 

f) determining motion by selecting a model according to constraints 
from one of translation, affine transformation and fundamental matrix, 
determining for each neighboring seeds a distance, and modifying the 
similarities between neighboring seeds according to the determined distance. 

48. A method for finding correspondence between portions of two images 
comprising the steps of: 

a) select two images inn and Im 2 , 

b) prepare for each pixel in Lmi a motion profile, 

c) assign a weight to each pair of neighboring pixels according to a 
normalized correlation between their motion profiles, 

d) perform a coarsening iteration to aggregate segments by 

i. clustering and re-estimation wherein clustering is achieved by selecting a set 
of seeds such that the remaining elements are strongly connected to this set, 
and defining the strength of association of a fine element to a coarse seed, 
and wherein re-estimation is achieved for each seed by determining the 
motion profile of the seed by multiplying the motion profiles of its children, 

e) continuing the coarsening iteration until a cluster is detected. 

49. A method according to claim 48, wherein the coarsening iteration determines 
segments adaptively. 
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50.A method according to claim 48, wherein the coarsening iteration uses 
bottom-up processing to disambiguate motion profiles. 



51. A method according to claim 49, wherein the motion of segments is varied 
according to a motion model with the level of scale determined by the amount of 
statistics that appears in each segment. 

52. A method according to claim 51, wherein the motion model varies from 
translation at fine levels, through affine and projective transformations in 
intermediate levels to 3D rigid motion followed by perspective projection 
(characterized by a fundamental matrix). 

53. A method according to claim 48, wherein the coarsening iteration produces a 
pyramid and motion is determined by combining constraints collected adaptively 
from different levels of scales in the pyramid. 

54. A method according to claim 48, wherein motion between the images is used 
to reconstruct a 3D structure of the scene depicted by the images. 

55. Apparatus for finding correspondence between portions of two images 
comprising a computer processor programmed: 

a. for selecting two images imi and im 2 , 

b. for preparing for each pixel in imi a motion profile, 

c. for assigning a weight to each pair of neighboring pixels according 
to a normalized correlation between their motion profiles, 

d. for performing a coarsening iteration by 

e. for clustering and re-estimation wherein clustering is achieved by 
selecting a set of seeds such that the remaining elements are 
strongly connected to this set, and defining the strength of 
association of a fine element / to a coarse seed /c, and wherein re- 
estimation is achieved for each seed by determining the motion 
profile of the seed by multiplying the motion profiles of its children, 
determining whether the seed is peaked or bar-peaked, 
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accumulating adaptively, moments (orders one to four) originated 
by peaked seeds, and accumulating separately, moments (orders 
one and two) by bar-peaked seeds, and 
f. for determining motion by selecting a model according to 
constraints from one of translation, affine transformation and 
fundamental matrix, determining for each neighboring seeds a 
distance, and modifying the similarities between neighboring seeds 
according to the determined distance. 



56. Apparatus for finding correspondence between portions of two images 
comprising a computer processor programmed: 

a. for selecting two images inn and im 2 , 

b. for preparing for each pixel in imi a motion profile, 

c. for assigning a weight to each pair of neighboring pixels according 
to a normalized correlation between their motion profiles, 

d. for performing a coarsening iteration to aggregate segments by 

ii. clustering and re-estimation wherein clustering is achieved by 
selecting a set of seeds such that the remaining elements are 
strongly connected to this set, and defining the strength of 
association of a fine element to a coarse seed, and wherein re- 
estimation is achieved for each seed by determining the motion 
profile of the seed by multiplying the motion profiles of its 
children, 

e. for continuing the coarsening iteration until a cluster is detected. 

57. Apparatus according to claim 48, wherein the coarsening iteration 
determines segments adaptively. 

58. Apparatus according to claim 48, wherein the coarsening iteration uses 
bottom-up processing to disambiguate motion profiles. 

59. Apparatus according to claim 49, wherein the motion of segments is 
varied according to a motion model with the level of scale determined by 
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the amount of statistics that appears in each segment. 



60. Apparatus according to claim 51, wherein the motion model varies from 
translation at fine levels, through affine and projective transformations in 
intermediate levels to 3D rigid motion followed by perspective projection 
(characterized by a fundamental matrix). 

61. Apparatus according to claim 48, wherein the coarsening iteration 
produces a pyramid and motion is determined by combining constraints 
collected adaptively from different levels of scales in the pyramid. 

62. Apparatus according to claim 48, wherein motion between the images is 
used to reconstruct a 3D structure of the scene depicted by the images. 
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